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English Placement Test Statistical Validation Report 

This is one in a series of reports describing and documenting the statistical validation 
evidence compiled in support of the placement tests used by the San Diego Community College 
District to recommend placement levels for matriculating students in English and Mathematics 
courses. Other reports concerning the SDCCD placement tests published by the Research and 
Planning office include the potential disproportionate impact of the English placement tests 
(Skills Testing and Disproportionate Impact, 1992); in-class surveys regarding placement 
accuracy and fairness (Faculty and Student Evaluations of Placement Practices in the SDCCD, 

1 990); alternate evidence gathered via matriculation surveys conducted over the last several 
years (SDCCD Matriculation Services Survey Report; Mesa College Matriculation Survey 
Report; Miramar College Matriculation Program Survey Report;, Student Perceptions of the 
San Diego Community Colleges); factors predicting success in English and mathematics 
courses, (Validating Placement Tests in the SDCCD, The Role of Biographical, Educational and 
Grading Variation); and reliability evidence of the placement tests (Research Brief on the 
Reliability of the APS Placement Tests). This report summarizes the content-related validity 
evidence and criterion-related validity evidence obtained both from the test publisher and locally 
to determine the validity of the inferences made from these tests about student ability and 
performance. 

Validity in Educational Measurement 

Central to the process of test validation is an understanding of what is meant by the 



phrase, "the validity of a test." The tests used in the SDCCD to place students in the English, 

math, or ESL curriculums are best viewed by all participants in the process as tools we use to 

make inferences about some ability or behavior. A noted psychometrician once observed that 

"one validates, not a test, but an interpretation of data arising from a specific procedure'." Thus 

the process of validation does not judge whether the test and the score it yields are valid, but 

rather the focus is on the interpretation of those test scores. Therefore the placement tests 

themselves are not validated, but instead we validate the inferences we draw based on the use of 

tests for placement in the curriculum. As stated by Popham: 

...there is no a simple on/off determination of whether a score-based 
inference is valid. Rather, we must make Judgements regarding the 
validity of score-based inferences. In order to make those Judgements 
as defensibly as possible, we assemble one or more types of evidence 
of validity^. 

The concept of validity in educational measurement is concerned with validity, of score 
based inferences. This is the operational definition of validity used in this study. 

Content Related Validity 

According to Popham, there are two basic strategies in assessing the content-related 
validity evidence of a test. One method attempts to incorporate suitable content on the test 
during the development of the instrument. Under this approach, the test developers use expert 
Judgement and professional advice to attempt to construct a test that adequately represents a 

^ Samuel A. Messick, "The Standard Problem; Meaning and Values in Measurement and Evaluation," 
American Psychologist, 30 (1975), 955-966 

^ James W. Popham, "Modem Educational Measurement: A Practitioner's Perspective . 2nd edition 
(1990); Angled Cliffs, NJ, Prentice Hall . 
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desired domain of content. For this study, evidence of this type was gathered from the technical 
manuals supplied with the APS tests that describe test development and technical data. 

The second method in gathering content-related evidence of validity is to examine the test 
and judge the representativeness of its content. Evidence for this was gathered during the initial 
selection of the APS tests by the SDCCD English faculty for use in placement. 

From the Publisher: 

According to the Standards for Educational and Psychological Testing^, validity 
evidence ..."demonstrates the degree to which the sample of items, tasks, or questions on a test 
are representative of some defined universe or domain of content." This is the operational 
definition used in this part of the study to assess the content-related validity of the English 
placement tests. The SDCCD reading and English placement system uses the Assessment and 
Placement Services for Community Colleges tests developed by the College Board under the 
auspices of the Educational Testing Service (ETS) of Princeton, New Jersey. These tests were 
drawn from the Comparative Guidance Program (CGP) which was a testing program developed 
in the late 1960's at ETS. The English, or Writing, portion of the test is the revised version of 
the CGP and the Reading placement test was introduced in 1984. The tests are hereafter referred 
to in this report as the APS Reading and Writing tests. There are instances at times in this report 
where the Writing test is also referred to as the English test. 

Evidence of content related validity was gathered through review of the technical 
manuals and materials included with the test materials provided by the College Board. This 



3 Standards for Educational and Psychological Testing. Prepared by a joint committee of the American 
Psychological Association, the American Educational Research Association, and the National Council 
on Measurement in Education (Washington, D.C.; American Psychological Association, 1985). 



evidence and descriptions of the test are described below. 



The APS Reading Placement Test 

The 25 minute APS Reading test contains 35 multiple choice questions based on eight 
reading passages. These selections, which are mostly written by contemporary writers use 
content drawn from the natural sciences, social sciences, and contemporary life. They vary in 
style and exemplify different types of writing: straightforward reporting, persuasive writing, and 
description. Questions are intended to measure the student's comprehension of main ideas and 
specific details and their ability to make inferences and extract the meaning of vocabulary in 
context. 

The test developer states that the placement scores are meant to differentiate among 
students who are "adequately prepared for academic work in college and those who may need 
developmental work, and they are useful in placing students in appropriate English courses." 
(College Board 1985; p. 6) Specific abilities tested and the related questions are shown in table 
1 . 

Table 1 



Reading Abilities Assessed and Corresponding Test Items on APS Reading 

Placement Test 



Specific Abilities in Reading 


Test Questions Used 


Understanding Main Idea 


1,8, 13,21,24, 25,28, 30 


Understanding Secondary Idea 


3,5,6, 9, 16, 20, 23,34 


Ability to Make Inferences 


12, 14, 16, 17, 22, 26, 29,31,33 


Understanding Vocabulary in Context 


2,4,7,10,11,15,27,32,35 



Source: College Board, 1984 

The APS Writing Placement Test 

The writing placement test contains 40 multiple choice questions intended to measure a 
student's ability to perform the "kind of writing usually required of students in colleges." 

(College Board, 1984, p 6). The test questions assess ability to recognize errors in grammar, 
usage, word choice, etc. The test appears to place emphasis on sentence structure, and clear 
expression in ideas and thoughts. There is no writing sample included in this assessment, the test 
relies solely on responses to a series of four part multiple-choice items. 

According to the College Board, a student scoring high on this placement test has a high 
probability of being able to write correctly and effectively. They cite prior research (Godshalk, 
1966) which indicated that scores on multiple choice questions similar to those used in the 
writing test are highly correlated to scores received by students on essay tests based on inter-rater 
reliability. The specific writing competencies assessed and relevant test items are shown in 
table 2. 



Table 2 



Writing Abilities Assessed and Corresponding 
Test Items on APS Writing Placement Test 



Specific Abilities in Writing 


Test Questions Used 


Sentence Recognition 


1, 18, 22,37 


Sentence Structure 


4,25, 28, 34, 35, 36 


Pronoun Problems 


3, 7, 20, 29, 33, 40 


Language and Style 


2,5, 13, 15,23 


Verb Problems 


8, 17 




5 



Logic 


10, 12, 16,21,24, 27,31,32,39 


Recognition of error-free construction 


6, 9, 1 1, 14, 19, 26, 30, 38 



Source: College Board, 1984 
Scores and Technical Data 



According to the publisher, the tests were scaled such that the score range extended from 
20 to 80 using standard scores, with an average of 50 and a standard deviation of 10. Thus the 
tests were originally scaled such that approximately two-thirds of the original norming group 
should have scores between 40 and 60. 

The placement test scores are intended to provide information to predict performance in 
college level courses in writing and reading, but do not provide diagnostic information. A longer 
test involving repeated samplings of a specific knowledge domain or competency would be 
required to make diagnostic referrals. As they are currently implemented, the placement tests are 
used solely for recommended referral to levels of English ranging from pre-collegiate to college 
level. Performance on groups of questions or competency clusters is not reported by the 
publisher. In reviewing the technical manual supplied with the placement tests it was noted that 
the test publisher recommends the use of standard scores, however the SDCCD does not use the 
standard scores but rather relies on raw scores. This was noted in an earlier report on 
Disproportionate Impact and that time the report recommended that the SDCCD convert the raw 
scores to standard score recommended by the test publisher. However the test score data 
analyzed for this report and to gather the reliability evidence still show the use of raw scores to 
make referrals. 

In the view of the English faculty and other faculty reviewing tests for use by the SDCCD 
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for placement into the English curriculum, the content of the APS tests was found to be the most 
closely aligned with the skill areas emphasized in the English department developmental and 
college level courses. Also the APS Placement tests have been approved for use in the 
California Community Colleges for placement by the State Chancellor's Office of the California 
Community Colleges. 

Criterion-Related Validity Evidence 

From the publisher: 

The technical manual and the APS tests themselves provide some evidence as to 
the content- and criterion-related validity of the APS placement tests. According to the technical 
manual supplied by the College Board to support the APS tests’, English faculty helped in 
"developing the tests and establishing their content validity" (p. 21). This assistance took the 
form of identifying the skills they thought necessary for success in English courses as taught in 
open-admission, post-secondary institutions. These instructors also approved all items used in the 
tests. The technical manual also provided information on the items used to address the content 
domain in community college English courses. 

The College Board sponsored validity studies that are summarized in the technical 
manual. These data are summarized in table 3 below. The validity studies they conducted relied 
primarily on correlations between test scores achieved by examinees prior to beginning college 



^ The College Board: The Assessment and Placement Services 
for Community Colleges Technical Manual. Princeton, NJ : 
Educational Testing Service. 
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coursework and grades earned at the end of the first or second semester. Correlations were 
obtained from 64 validity studies, each of which had at least 100 students. The criterion variable 
was the final grade earned in the English course. 



Table 3 

Correlations of Test Scores and Grades in 
English Courses as Reported by the College Board 



Course 


Predictor 


Median 


Highest 


Lowest 


English 


Reading 


.28 


.51 


-.01 




Writing 


.32 


.55 


.03 




Reading 
and Writing 


.35 


.57 


.04 



Although the correlations submitted by the College Board appear to have a moderate 
relation to the criterion variable, there are critical questions that were not addressed in the 
summary presented in the technical manual. For example, correlations between final grade and 
test by course level (pre-collegiate and college) were not given. Also, were students who 
dropped or withdrew late in the course included in the study? Did they treat withdrawals as 
failures and assign them a value of "zero" for analysis purposes or were these students not 
included in the study? Were students taking the class for Credit/Non Credit included? If so, 
what value was assigned a credit grade? A non credit grade? Another problem not discussed in 
the Incomplete grade notation. Did the INC notation also covmt as a failure or did the researchers 



drop these students from the study? The problem of what has come to be called the 
"troublesome W" in validation studies conducted by research offices in the two year colleges is 
not addressed in the APS study. For example, the high proportion of W notations (approximately 
16-18%, depending on the terms analyzed) appearing in the student transcripts for English and 
mathematics courses here in the SDCCD is a potentially confounding factor in interpreting the 
correlation between final grade and placement test score. The W grade is troublesome because of 
the uncertainty about what it represents. In some instances, the W could be considered a non- 
successful grade if students who are performing below a certain level withdraw from the course 
to avoid a low grade. On the other hand, a student may be doing well in a course and drop late in 
the term for reasons unrelated to academic ability or commitment to college attendance. These 
unanswered questions regarding the study design and methodology make it difficult to compare 
- these findings with local validation studies using similar criterion-related validity evidence. 

For the purposes of this study two dependent variables are used most often, one 
including, and the other excluding the troublesome W. In several of the tables and regression 
models presented in this report, the W grade has been included in a variable called Modified 
GPA (MODGPA). This definition codes an A grade as equal to four points, a B grade as equal 
to three points, a C grade as equal to two. points, and D, F, W, INC, as equal to one point. 
Traditional GPA as a dependent variable is also used in several of the tables and analyses for this 
validation study. In cases where it was important to maintain adequate sample sizes, and for 
informational purposes, the modified GPA was included in the analyses. This was done both to 
analyze the extent to which the W is a measure of academic performance, and to provide the 
college academic leadership with information on the proportion of W grade notations given 
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during the term. 



Local Studies 

The relationship between APS Reading and Writing placement scores and course 
performance has been studied locally. This is the third report that includes an examination of the 
relationship between APS Placement tests and course grades in English. The first report used an 
in-class survey to ask students if they were placed accurately, that is, did they have sufficient 
skills to achieve at least a 'C grade in the course? In this same study, faculty were also asked to 
indicate the proportion of students placed in their class who belonged in a higher level course, a 
lower level course, or were placed accurately. This study found that overall, both students and 
faculty felt that the APS Placement tests resulted in a high degree of placement accuracy^. The 
second study which included an examination of the relationship between APS Placement scores 
and course performance examined several factors that predicted final course grade in English and 
mathematics courses in the SDCCD^. This report showed local statistical criterion related 
evidence of the APS Placement tests through the use of correlational methods and regression, 
and also identified several other important variables that predict final grade, among them 



^ Student and Faculty Evaluation of Placement Results : 
Results of In-Class Surveys of Faculty and Students Regarding 
SDCCD Placement Practices and Results. SDCCD Research and 
Planning, Spring, 1991 

^ Matriculation Report: Validating Placement Tests in the 

San Diego Community College District : The Relationship of 
Placement Test Scores, Biographical Data, and Grading Practices 
to Course Success, SDCCD Research and Planning, 1994 
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instructor grading variation. 

Additional alternative evidence addressing fairness, accuracy, and general satisfaction 
with assessment and placement in the SDCCD has been gathered using survey data. For 
example, the Matriculation Services Satisfaction Survey conducted in 1 992 asked students to rate 
their satisfaction with English and mathematics assessment. Although the results varied slightly 
by college, overall approximately 75% rated the assessment process as "Excellent or Good." 
These results were not found to differ appreciably by ethnic, racial, or gender grouping. 

Another survey conducted districtwide in the spring, 1993 term found similar results. Of the 
students surveyed who recognized and used assessment and placement services, almost 80% 
were satisfied with these services.^ As with the survey conducted prior to the 1993 survey, there 
were not large differences between racial, ethnic, and gender groupings in response to this 
question of satisfaction with assessment and placement. 

Specific college surveys conducted by the Research department have tended to confirm 
the evidence of the fairness of assessment and placement districtwide. For example, a study 
conducted by the SDCCD Research Office for the Mesa College Matriculation office found that 
a clear majority of students who took the placement tests in English and mathematics agreed that 
they were placed in the correct level of English (81%) and mathematics course (76%). A similar 
proportion (82.2%) also agreed that they understood the explanation they received about the 

* Matriculation in the SDCCD: Equity and Student 

Satisfaction with Matriculation Services, SDCCD Research and 
Planning, 1993 

® Survey ' 93 : Student Perceptions of SDCCD Educational 

Services, SDCCD Research and Planning, July, 1993 



results of the assessment and placement tests. Further, students who indicated that they 
understood the explanation of their assessment results were significantly more likely than 
students who did not to say that they were enrolled in the correct level of English (87% 
compared with 59%). A similar finding obtained for mathematics course placement. This is 
evidence that counseling, where possible, is performing an essential service in the clear 
explanation of the meaning of placement test results®. This is a positive indication of the success 
of integrating assessment results with student consultations. Students who have been explained 
the context, meaning, and purpose of assessment and placement as part of an overall educational 
planning process are far more satisfied with English and mathematics assessment. 

Evidence as to the accuracy, fairness, and satisfaction with the assessment process was 
also gathered using a Matriculation Services Survey at MirEimar College’. Students were asked 
to respond to a series of statements regarding various Matriculation services at the college using 
a Likert type scale. Several statements focused specifically on the assessment and placement 
component. Student responses suggested a high degree of satisfaction with the assessment 
process for accuracy and fairness. For exEimple, statement #9 on the survey, "My test scores 
helped me to enroll in the right level of English class," elicited a clear majority of affirmative 
responses. Almost 90% either agreed strongly, or agreed with this statement. No significant 
differences from this pattern were noted among gender or ethnic groups. Older students (21-30) 



^Student Satisfaction with Matriculation Services: San 

Diego Mesa College, SDCCD Research and Planning, July, 1993. 

’Student Satisfaction with Matriculation Services: San 

Diego Miramar College, SDCCD Research and Planning, July, 1993. 
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tended to agree with this statement (89%) more than younger (17-21) students (84%). Similar 
results were found for the statement regarding the role of test scores in enabling students to enroll 
in the right level of mathematics course. A clear majority (78%) either agreed or strongly agreed 
that assessment test scores in math helped them enroll in the right level of math class. Slight 
differences were found in the pattern of responses between men and women with men showing 
higher levels of agreement (80%) than women (75%). The view that assessment test scores 
helped them enroll in the right level of math class was expressed at higher rates by students 
between the ages of 21-30 years (82%) than by those aged 17-20 years (68%). The analysis 
found that responses to this question by race and ethnicity did not differ in a practical sense from 
the general pattern. As was the case with the Mesa college respondents, understanding the 
results of placement testing was found to be an important moderating variable on the perception 
of the fairness and accuracy of testing. 

Criterion-Related Validity Evidence: The Current Study 

In addition to the evidence described above, the statistical relationship between the APS 
Reading and Writing Placement test scores and course performance was also studied locally 
using methodology recommended by the State Chancellor's Office of the California Community 
Colleges in the Matriculation Local Research Options handbook. Both tests were shown to be 
significantly correlated with English course performaiice at the SDCCD colleges. In addition, 
regression analysis identified several variables which serve as additional predictors of 
performance. The following section describes these results. 

Sample. The students included in the primary analyses for this study were all students 
enrolling in the SDCCD who were assessed during the fall, 1994 testing period and subsequently 
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enrolled directly in an English course and received a grade notation on their transcripts. Student 
records were matched using the computerized assessment and placement system used in the 
testing centers and the Research and Planning student extract files which contain demographic 
data, courses taken, and final grade received. More than four thousand cases were available for 
analyses. Depending on the criterion variable and level, of course analyzed, the number of cases 
in particular analyses presented in this report does vary, but remains sufficiently high to maintain 
reliability. Descriptive statistics for the sample are included in the appendix to this report. 



Descriptive Statistics and Correlation Analysis. The correlation between the APS 
Reading, APS Writing, and the combined APS Reading and English Placement test scores and 
grades in all English courses during the fall, 1994 term are presented in table 4 below. 

Table 4 

Descriptive Statistics for Reading, Writing, and Grades in all English Courses in the 

SDCCDFall, 1994 



Variable 


Cases 


Mean 


Std Dev 






Score 




Reading Test 


4322 


17.8 


7.3 


Writing Test 


4322 


22.0 


7.0 


Combined 


4321 


39.9 


13.3 


Reading and 
Writing Tests 








MODGPA 


3834 


2.1 


1.0 


GPA 


3022 


2.4 


1.1 



The correlations of reading and writing placement test scores with the dependent 



variables (both traditional GPA and modified GPA) are given below in table 5. 



Table 5 

Correlation of APS Reading and Writing Tests with Outcome Measures 

For All Levels of English 



Criterion 


APS Reading 


APS Writing 


Combined 


GPA* 


.1993 


.2384 


.2360 


(N) 


( 2503) 


( 2504) 


( 2503) 


Probability 


P= .000 


P= .000 


P= .000 


MODGPA** 


.1462 


.1953 


.1835 


(N) 


( 3227) 


( 3228) 


( 3227) 


Probability 


P= .000 


P= .000 


P= .000 



Note: 1-tailed Significance Levels are shown. *GPA is a five point scale with A=4, B=3 , C=2, 

D=l, F=0 all other notations are deleted from the analysis. ** MODGPA is a five point scale 

similar to GPA but includes D, F, W, NC as a score of 1. 

The correlation between the combined APS Reading and English Placement test scores 
and course performance in English 101 was .24 for GPA based on a five point scale (A=4, B=3, 
C=2, D=l, F=0), and .22 using a criterion variable called Modified GPA. This dependent 
variables codes final grade notations as A=4, B=3. C=2, D, F, W, NC=1 . For the APS 
Reading test, the correlation with final grade (GPA) was .18 and for APS Writing, the coefficient 
was .25. A similar set of coefficients was found when examined for English 101 courses only as 
shown in table 5 below. In each case the relationship between final grade and the test scores are 
statistically significant although writing demonstrates a consistently stronger relationship with 
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the dependent variables than does the reading test. 

It has been noted by other analysts conducting validation studies including the state 
chancellor's office that moderate or low correlation coefficients does not necessarily mean the 
measures used to predict performance are not related to the dependent variable of final grade. 
Review of test validation literature and prior analyses by the institutional research office 
suggested several technical constraints that may serve to weaken or lower the observed 
correlations. These include: 

( 

1 . Restriction of range in the criterion variable of final grade, 

2. Restriction of range on the predictor due to use of an existing cut score to place 
students! and, 

3. The instability of the criterion measure of final grade due to grading variation. 
Investigation of these constraints in prior analyses* found that because the validation studies 
conducted here in the SDCCD have relied on retrospective data obtained on an existing 
assessment and placement system which in effect pre-sorts students, the observed correlation 
coefficients may be lowered. This truncation of distributions into different course levels restricts 
the range of abilities and outcomes available for analyses. An application of restriction range 
does improve the relation between the test scores and the dependent variable as shown below in 
table 7. But this statistical artifact is of little value in validating a particular score on a test with 
the outcome measure of final grade. 



^Matriculation Report; Validating Placement Tests in the 
San Diego Community College District, The Relationship of 
Placement Test Scores, Biographical Data, and Grading Practices 
to Course Success, SDCCD Research and Planning, Spring, 1994 
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Prior analyses conducted by the research office also found several variables correlated 
with final grade using regression analyses, including variation in the dependent variable of final 
grade^. This report found several demographic and educational variables as significantly 
contributing to the prediction of final grade in several levels of English course. These included 
test scores, high school GPA, years out of school, employment hours planned, and 
environmental variables that may be instructor related. 



Table 6 

Correlation of APS Reading and Writing Tests with Outcome Measures 

For English 101 



Criterion 


APS Reading 


APS Writing 


Combined 


GPA* 


.1797 


. 2521 


.2418 


(N) 


( 796) 


(796) 


(796) 


Probability 


P= .000 


P= .000 


P= .000 


MODGPA* * 


.1684 


.2470 


.2299 


(N) 


( 1085) 


( 1085) 


( 1085) 


Probability 


P= .000 


P= .000 


P= .000 



Note: 1-tailed Significance Levels are shown. 

*GPA is a five point scale with A=4, B=3, C=2, D=l, F=0 all other notations are deleted from the analysis. 
MODGPA is a five point scale similar to GPA but includes D, F, W, NC as a score of 1 . This model has been used 
in other validation studies conducted in the community colleges in California 

Table 7 below shows the correlation of test scores with final grade obtained in three 

levels of English course. An application of restriction of range is also shown. This correction 

uses the smaller standard deviation found in cases where students have been distributed due to 
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(ibid. ) 



the application of an existing testing scheme that in effect pre-sorts students thus restricting the 
range of available scores. 



Table 7 

Correlations, both Corrected and Uncorrected, 
of APS Reading and Writing Test Score with GPA by Level of English 



Course 

Level 


Transfer Level 
(101 and 
above) APS 
Reading 


One Level 
Below Transfer 


Two Levels 
Below Transfer 


Placement Test 


Pearson r 


Pearson r 


Pearson r 




(Corrected r) 


(Corrected r) 


(Corrected r) 


APS Writing 


.25 


.30 


.29 




(.36) 


(.41) 


(.41) 




P= .000* 


P= .000 


P= .000 




N=626 


N=895 


N=315 


APS Reading 


.18 


.20 


.11 




(27) 


(.30) 


(.17) 




P=000 


P=000 


P=000 




N=626 


N=895 


N=315 


*Probabilities basec 


on corrected Pearson correlation only 



The data in table 7 indicate that the correlations between the placement test and the 
dependent variable improves with the application of a correction for restriction of range. In the 



case of the APS writing test, the correlation surpasses a .35 correlation which has been regarded 
by state officials as indicative of sufficient criterion-related validity evidence forjudging the 
statistical validity of the placement tests to a criterion of final grade. In the case of the reading 
test, the application of the correction for restriction of range, although improving the observed 
correlation coefficients substantially, does not achieve the targeted .35, although the correlations 
are positive and statistically significant. Graphical depictions of the relationship between scores 
on the APS Placement tests and success in English are shown in figures one and two below. 

The relationship between 



APS Reading score and English 101 
course performance is shown in 
figure 1 . The proportion of 
students passing and failing English 
101 is shown for different ranges of 
the APS Reading Placement test. As 
APS Reading Placement Test score 
increases, the proportion of students 



Figure 1: APS Reading Score 

by English 101 Success 
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passing tends to increase. Passing 
English 101 is thus positively 

associated with higher APS Reading Placement test scores. When analyzed for all English 
courses, a similar trend was foimd as with English 101 course success. Using the above score 
ranges on the APS Reading Placement test, the proportion of students passing their respective 
English course is approximately 50% at the lowest ranges, to approximately 69% at the upper 
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range. 



The relationship between APS 
Writing Placement test scores and 

English 101 course performance is Figure 2 ‘. APS Writing Score 

by English 101 Success 



I I 1 I I 

Stoll 12 to17 18to23 24to29 30to35 36to40 

associated with higher APS Writing 
Placement Test Scores. This same 
pattern of success is also found for all 
students in English courses. When analyzed for all English courses using the same score ranges 
identified in figures one and two, the proportion of students passing their respective English 
course based on APS Writing Placement test score ranges fi-om approximately 48% at the lowest 
level to almost 70% at the highest levels. This is further evidence of the positive relationship 
between APS Writing Placement test and English course success. 

Regression Analysis: English 101. To determine suitable predictors for use in a 
placement system that included placement test scores, multiple linear regression was used. The 
purpose of rriultiple regression analysis is to identify a group of student variables which 
optimally predict course performance. Potential predictor variables were those which were 
significantly correlated with course grade. These included high school GPA, Years of English in 
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High School, Grade in Last English Course, Years Since Last Attended School, and APS 
Placement Test Score. Multiple regression analysis showed that the APS Reading and Writing 
Placement test scores are significant predictors of English 101 course performance, even when 
other relevant variables are included in the model. For this analysis, the dependent variable was 
English course grade. This was selected because the assessment test's purpose is to predict 
success in English. If a student's chance of succeeding in English 101 is highly uncertain, the 
student is directed to enroll one or two levels below freshman composition in the reading and 
writing tracks that are foundation courses for freshman composition, depending on score 
achieved on both placement tests. For this analysis, withdrawals were included as non-successes 
to maintain a reliable sample size. 

Analysis was performed using the stepwise regression procedure of the Statistical 
Package for the. Social Sciences (SPSS). The summary table is shown below. The variable 
labeled Q9 is taken from the assessment survey and refers to years out of school for the 
respondent, highgpa is high school grade point average, and Q1 1 refers to the grade the student 
received in his or her last English course. 



MULTIPLE 
Equation Number 

Summary table 



Step 


MultR 


Rsq 


F (Eqn) 


SigF 


Variable 


Detain 


1 


.2291 


. 0525 


43.723 


.000 In: 


TOTRDEN 


. 2291 


2 


.3158 


. 0998 


43 . 660 


.000 In: 


HIGHGPA 


. 2176 




REGRESSION **** 

1 Dependent Variable . . MODGPA 
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3 

4 



.3858 

.3984 



.1488 
. 1588 



45.871 .000 In: Q9 

37.083 .000 In: Qll 



.2330 
- . 1148 



Standard Error of Est . 1.04481 Signif F Change 



.0024 



DF 



Analysis of Variance 

Sum of Squares Mean Square 



Regression 

Residual 



4 

786 



161.92405 

858.02032 



40.48101 

1.09163 



F = 37.08313 



Signif F 



. 0000 



APS Placement test scores entered the regression equation first, indicating that these 
combined scores were the strongest available predictor of English 101 course grade. The next 
variable to enter the equation was high school GP A. Additional variables to enter the equation 
from those specified for the model included length of time out of school (coded as l=Still 
Enrolled, and 6= More than 10 years), and grade received in last English course (coded as 1=A, 
and 5=F, thus the negative association). Together the four variables combined for a multiple 
correlation of approximately .40 and a r-squared of approximately .16 indicating that 
approximately 16% of the variance in freshman composition is predicted by these four variables. 
The standard error of the estimate was 1.04. The final regression equation explained a highly 
significant proportion of the variance, F=38.08, (p<. 00005). 

Although APS Placement test score. High School GPA, Years Since Attended School, 
and Grade Received in Last English Course entered the stepwise regression model because of 
their significant relationship to the dependent variable of modified GPA, a practical useful 
approach to developing a multiple measures placement scheme militated for a simpler model. 



There were several reasons for this. Chief among them had to do with response rates on the 
.assessment survey form. For example, it was found that approximately twelve to fifteen percent 
of respondents from that term did not answer the high school GPA question. A somewhat higher 
proportion did not respond to the question about grade in last English course. Because multiple 
regression employs listwise rather than pairwise deletion of missing data, the proportion of 
missing data from these questions greatly reduces the number of cases available for study . In a 
practical sense, the more questions are relied upon to develop a placement recommendation, the 
greater the likelihood that a student will not have responded to one of the needed questions. This 
would result in greater time spent in obtaining the information from the student or referral to a 
counselor. In addition, a simpler model would be easier to explain and communicate to students, 
faculty, and counselors involved in the placement process. A simpler model would also facilitate 
modification of cut scores by limiting the number of considerations for modification to a 
minimum while still maintaining reliability in placement. Therefore, a second multiple 
regression was conducted that included only the first two variables to enter the equation 
described above. The summary table is shown below. 



MULTIPLE 



REGRESSION 



Dependent Variable . . 



MODGPA 



Summary table 



Step MultR 

1 .2265 

2 .2984 



Rsq F(Eqn) 

.0513 57.036 

.0891 51.523 



SigF 
.000 In: 
.000 In: 



Variable Detain 
TOTRDEN .2265 
HIGHGPA .1948 
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25 



standard Error 



1 . 08011 



Signif F Change .0000 



Analysis of Variance 



DF 

2 



Sum of Squares 



Mean Square 



Regression 

Residual 



1054 



120 . 21783 
1229 . 64215 



60 . 10891 
1 . 16664 



F 



51 . 52295 



Signif F = .0000 



The regression model that included high school GPA and the combined APS Placement 
test scores yielded a multiple R of .29 and a r-squared of .09. Although the proportion of 
variance explained was lower than the first model which included two additional variables, the 
sample size was much greater due to a lower number of missing responses, and the resulting 
proportion of the variance explained was still highly significant statistically. (F=51.53, 
(p<.0005). 

APS Placement test score and high school GPA therefore reliably predicted grades in 
freshman composition. As stated above, it was determined that a straightforward method of 
calculating the combined predictor variable was needed. A simple formula (as opposed to a 
regression equation with decimal weights assigned to the variables in the model and had the 
advantage (in addition to those described above) in that individuals involved in the placement 
decision, counselors, faculty, and students, can readily calculate it. Of several simple 
formulas studied, the most effective and straightforward was found to be a combination of 
score ranges on both the APS reading and writing placement tests and high school grade point 
average from the assessment survey forms. To include high school grade point average which 
is a seven point scale with the APS Reading and Writing placement tests which are scaled from 



1-35 and from 1-40, respectively, the placement tests were recoded to correspond to the seven 
point high school grade point average scale used on the assessment survey form. The 
recategorization of the placement tests into two scales of seven points each followed the 
scheme identified below. 



Table 8 

APS Reading and Writing Placement Scores and High School GPA Scale and 
Recategorized Scores based on Ranges for Each Variable 



APS Reading 


Reading 

Scale 


APS Writing 


Writing 

Scale 


High School 
GPA 


High School 
GPA Scale 


1-5 


1 


1-4 


1 


0.0-0.9 


1 


6-10 


2 


5-11 


2 


1.0-1.4 


2 • 


11-15 


3 


12-17 


3 


1.5-1. 9 


3 


16-20 


4 


18-23 


4 


2.0-2.4 


4 


21-25 


5 


24-29 


5 


2.5-2.9 


5 


26-30 


6 


30-35 


6 


.3. 0-3. 4 


6 


31-35 


7 


36-40 


7 


3. 5-4.0 


7 



This recategorization of the placement test scores thus parallels the high school GPA 
scale obtained from the CAPP survey form given during assessment. The recoding of the APS 
Reading and Writing placement tests results in two seven point scales, one from the APS 
Reading Placement test and the other from the APS Writing Placement Test. 

High School GPA. At the time of assessment, students are asked to answer a survey 
regarding their educational goals, plans, and past academic performance. Question number 12 
on the survey asks students to report their high school GPA using a seven point scale. The 
scale is reproduced in the preceding table and was the same scale used in the regression 
models to estimate final grade in English. This scale ranges from one to seven. This seven 



This seven point scale is added to the seven point Reading scale and the seven point Writing 
scale. Thus: PREDICTOR=APS WRITING SCALE (7) + APS READING SCALE (7) 

+ HIGH SCHOOL GPA SCALE (7) =21 
In this formula, the APS Writing Scale and the APS Reading Scale is derived 
according to the scheme described in the above table, and added to the High School GPA scale 
to create a 21 point scale. Although as stated earlier, this reduced regression model has a 
lower r-squared than the full model considered initially, the resulting multiple R is still highly 
significant and is more parsimonious than the full model. Moreover, the standard error of the 
estimate is nearly identical to the full model (1.08). Also, inspection of the correlation matrix 
generated by the SPSS regression routine suggests a weak relationship between the combined 
APS Reading and Writing Placement tests and high school GPA (r= .042, p = . 1 17). The new 
predictor (titled RIEIHSGP in the subsequent analyses), shows a relatively normal distribution 
as shown by figure 3. Descriptive statistics for the new English predictor are given below. 
The mean for the full sample is 13.3 with a standard deviation of 2.85, the median is 13, and 
the modal value was also 13. This suggests that the model has an approximately normal 
distribution with a minimum skew either positively or negatively. The observed standard error 
is .045. This relatively narrow standard error suggests that new predictor is a reliable scale. 
This is confirmed by inspection of figure three below. Quartiles are also given for the 
predictor. 

Figure 3: Predictor Descriptive Statistics 



Mean 


13 ,313 


Std err 


. 045 


Median 


13 


. 000 


Mode 


13 . 000 


Std dev 


2 . 851 


Variance 


8 . 


130 


Range 


16 . 000 
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5.000 


Maximum 


21 


.000 
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Percentile Value. Percentile Value Pejccentile Value 



25.00 11.000 50.00 13.000 75.00 15.000 

Valid cases 4007 



Figure 3: English Predictor Distribution 
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Figure 3 

Distribution for English 101. For freshman English the predictor still shows a 
relatively normal distribution as shown by figure four (a mean of 15.5 and a standard 
deviation of 2.4) for the sample that took the English placement tests and enrolled in an 
English course in the fall, 1994 semester. The measures of central tendency suggest an 
approximately normal distribution with only a slight positive skew. The standard error for the 
freshman English sample is .063. Although slightly higher than the full sample, this standard 



error is still relatively narrow and may be slightly higher due to attenuation of range by 
including only the freshman English group. 



Figure 4: Predictor Descriptive Statistics for Freshman English 
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Figure 4 



Referral to the English Curricular Tracks 

If a student scores less than the recommended cut score of 15 on the English 101 
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Predictor scale, then they are referred to one or both of the reading and writing tracks that are 
below the freshman composition level. To have the placement system parallel the bifurcated 
English curriculum that have separate tracks for reading and writing below the freshman 
composition level, two scales from the predictor scale were broken out. This was done to 
determine placement within each track. Within these tracks, students must be referred to one 
level below freshman English (English 51 or English 56 or both) or two levels below freshman 
English (English 50 or 55 or both). For the both tracks a 10.5 point scale was developed with 
high school GPA recategorized into one-half of its original value and added to the existing 
seven point placement test scales described earlier. In this way high school GPA would still 
be weighted one-third for the new Reading and Writing Predictors. Thus the Reading and 
English Predictors for below freshman composition = Reading Predictor =(HSGPA (7 *.5) + 
Reading Scale(7) = 10.5. Similarly for Writing the scale is the same with HSGPA comprising 
one-third of the recommended scale. Descriptive statistics for these scales for the below 101 
group (i.e., below 15 on the English 101 predictor) show that for the Reading Predictor the 
average score is 5.6 with a standard deviation of 1. The standard error for the Reading 
Predictor is .021. For the Writing Predictor the mean score is 6.0 with a standard deviation 
of 1 .0 and a standard error of .20. Both the Reading and Writing predictors for below the 
freshman composition level show approximately normal distributions with a narrow standard 
error. The distributions are shown in the next two figures. 
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Figure 5 
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Figure 6 



Both the Reading and Writing Predictor scales show criterion- related validity evidence 
similar to that found for the English 101 predictor scale. This is shown in the correlation 
matrix below. Both the Reading and Writing Predictors for the levels below freshman 
composition are statistically significant. It is expected that a corrected Pearson correlation 
coefficient would boost these observed correlation substantially. 



Reading Writing 
Predictor Predictor 



GPA .1458 .1735 

( 1516) ( 1516) 

P= .000 P= .000 
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Criterion-Related Validity Evidence Summary 



The evidence gathered and presented in the preceding analyses suggests that the APS 
Reading and Writing Placement tests therefore show acceptable criterion-related validity. The 
hypothesized English 101 Predictor scale based on a combination of Reading and Writing 
placement tests and high school GPA also show acceptable criterion-related validity. In 
addition, the sub-scales of the English 101 Predictor for both Reading and Writing tracks also 
show acceptable levels of criterion related validity. The statistical evidence presented in the 
correlation matrices and crosstabulations is also supported by alternative evidence gathered by 
special Matriculation surveys of students and faculty conducted by the Research department 
which also provides criterion-related validity evidence of the validity of the APS Placement 
tests to course success. 

Reliability Evidence 

One indicator of the reliability or stability of the placement tests is a comparison of 
scores over time and across settings. To provide evidence for the reliability of the placement 
tests, four samples of the SDCCD testing population were analyzed for various semesters from 
the period of fall, 1988 through fall, 1995. The tables that follow below present data from the 
fall, 1988, fall, 1990, fall, 1994, and fall, 1995 tested populations to enable comparisons of 
the scores for each of the colleges over a seven year period. The similarity of scores over 
time suggest that the tests are generally reliable. A prior study conducted by the SDCCD 
Research office analyzed and compared the norming groups used by APS with the SDCCD 
testing population to determine if there were significant differences between the two groups 
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when examined by scores. This study determined that although there slight differences 



between the two groups when measured by average scores, it was shown that there were 
significant differences between the norming group used by APS and the SDCCD tested 
population. 

Scores over Time, 1988-1995. Table 9 below shows test scores from four different 
testing samples over a seven year period. Means, standard deviations and sample sizes are 
given to enable comparison of test scores during this period. This analysis of four fall 
semesters provides evidence of APS test reliability with the SDCCD student population 



because of the approximate similarity of test scores over time. 



Table 9 

Means, Standard Deviations and Distribution of Placement Scores 
SDCCD Tested Population 



Year 


City 


City 


Mesa 


Mesa 


Miramar 


Miramar 


ECC 


ECC 


SDCCD 


SDCCD 


Reading 


Writing 


Reading 


Writing 


Reading 


Writing 


Reading 


Writing 


Reading 


Writing 




Mean 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 


Mean 




(SD) 


(SD) 


(SD) 


(SD) 


(SD) 


(SD) 


(SD) 


(SD) 


(SD) 


(SD) 




N 


N 


N 


N 


N 


N 


N 


N 


N 


N 


1988 


19.0 


23.0 


22.0 


25.6 


21.0 


24.7 


13.2 


17.9 


20.0 


23.5 




(7.9) 


(7.2) 


(6.3) 


(6.2) 


(7.3) 


(7.3) 


(7.2) 


(7.7) 


(7.7) 


(7.2) 




2671 


2669 


692 


692 


1110 


1110 


201 


201 


4676 


4676 


1990 


18.2 


21.3 


20.4 


23.8 


19.2 


23.4 


17.4 


20.1 


19.3 


22.7 




(7.3) 


(7.0) 


(6.8) 


(6.8) 


(6.8) 


(6.7) 


(7.3) 


(7.0) 


(7.0) 


(6.9) 




1122 


1122 


1463 


1463 


470 


470 


221 


221 


3277 


3277 


1994 


17.0 


19.7 


18.7 


23.4 


17.3 


22.8 


15.3 


18.7 


17.8 


22.0 




(7.1) 


(6.8) 


(7.3) 


(6.8) 


(7.4) 


(7.2) 


(6.5) 


(6.0) 


(7.3) 


(7.0) 




1542 


1543 


2068 


2068 


716 


716 


118 


118 


4326 


4327 


1995 


18.5 


21.7 


19.4 


24.0 


19.9 


22.8 


* 




19.2 


23.0 




(7.6) 


(7.1) 


(7.2) 


(6.6) 


(7.3) 


(7.2) 






(7.4) 


(6.9) 




1209 


1210 


1398 


1398 


590 


590 






3197 


3197 



^“Skills Testing and Disproportionate Impact: An Analysis 

of the Reading and Writing Test Performance of Students in the 
San Diego Community College District. SDCCD Research Office, 
Spring, 1991 
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♦ Data for ECC not available fall 1995 



When examined for each of the colleges and ECC in 1988, scores on the reading test 
appear generally similar with the exception of the college courses offered at ECC. For City 
College, the average score on the reading test was approximately 19, with a standard deviation 
(SD) of approximately eight. For Mesa College, the average score was slightly higher at 22.0, 
with a standard deviation of approximately 6.3. Reading scores for the Miramar tested 
population was approximately midway between City and Mesa Colleges at about 21.0 with a 
standard deviation of 7.3. Although differences exist among the three college campuses, they 
lie within the error of measurement reported by the test publisher and confirmed by this study. 
At the Educational Cultural Complex, scores on the reading test were significantly lower 
(p < .05) than the mean scores reported by the other three sites. 

A similar pattern of test performance can be observed with respect to average scores 
on the Writing test. As indicated in table 9 above, the City College sample had slightly lower 
scores on the APS Writing test than did either the Mesa or Miramar samples. As with the 
reading test, the differences while statistically significant (primarily due to large sample size), 
are of limited practical significance. For the 1990 sample, means and standard deviations by 
campus were again computed and analyzed. Although there appears to be a slight downward 
trend, the differences between the two terms do not appear to be important. The 1994 sample 
appeared to suggest a continuing trend downward in average test scores, however, by 1995 
this trend had started to move back to the 1988 levels in both Reading and Writing. The 
average scores in 1988 was 19.6 for the Reading test and 23.5 for the total SDCCD. The 
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scores in fall 1995 were 19.2 for Reading and 23.0 for Writing. Comparison of these data 
provide additional evidence of the reliability of the test scores from year to year, for four 
different testing cohorts. It shows that over the seven year period, the scores achieved on the 
APS placement tests by successive testing cohorts are generally similar. This finding obtains 
despite expressed concerns by many in the college community over the perceived declining 
academic skills of students coming to the colleges. Thus analysis of scores obtained on the 
APS Placement tests over a seven year period are within one or two points of the average 
scores observed over a seven year period for four different cohorts. It also appears that the 
reading and writing abilities of college freshman as measured by the placement tests has 
remained relatively stable over the years analyzed. 

Reliability of Predictor. The reliability of the predictor was analyzed by comparing the 
English predictor scores of two successive testing cohorts. 



Table 10 

Fall, 1994 and Fall, 1995 Predictor Means 



Term 

N 


Predictor Mean 


Standard Deviation 


Standard Error 


Fall, 1994 


13.3 


2.85 


.045 


4007 








Fall, 1995 


13.6 


2.89 


.030 


4655 









The distributions for the two terms analyzed for reliability evidence appear very similar with 
relatively narrow standard deviations. The following two figures illustrate the distributions of 
the predictor means for two successive fall testing cohorts. Figure 7 below shows the 




37 



35 



distribution of the 1994 Predictor. Figure 8 shows the fall, 1995 distribution of the Predictor. 
These distributions further confirm the stability of the predictor scale. 



Figure 7 

Fall, 1994 Predictor Mean and Descriptive Statistics 
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Figure 8 

Fall, 1995 Predictor Distribution and Descriptive Statistics 




In summary, both the APS Placement scores and the derived prediction equation show 
stability over time. The means of the scores for two successive terms of the predictor are 
remarkably similar. 

Test Bias and Disproportionate Impact 

From the Publisher: The technical manual does not specifically address disproportionate 
impact issues related to the APS placement tests. They focus primarily on the fact that 
minority students were approximately one-fourth of the norming sample and thus the norming 
data reflect the characteristics of a large proportion of ethnic minorities. They also assert that 
because the tests were normed on a representative sample of community college students, that 
the tests are an accurate and fair measures of English ability and competence to challenge 
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college level course work. No separate discussions of the impact of the tests on students by 
age or gender groupings were in the technical manual accompanying the tests. 

Local Studies. Local studies suggest that the APS English and Reading Placement tests show 
little disproportionate impact. A disproportionate impact study conducted locally indicated 
that although certain groups of students performed more poorly on the placement tests, that 
these differences were largely attributable to educational background and differences in 
English communication skills For example the research found differences among the 
various ethnic and linguistic groupings with respect to the grade they reported receiving in 
their last English class. White students reported higher percentages of A and B grades in their 
last English class compared with all other students. This was viewed as evidence that 
differences in the average scores achieved by non-white students could, at least in part, be 
attributed to prior achievements in English courses. In addition, there were differences found 
in the high school GPA of certain non-white groups such as Black and Latino students 
compared with White, Pacific lislander, and Asian students with the latter groups indicating 
higher GPA’s in high school. This information helped to account for the observed differences 
noted in the performance of these student groupings on the APS Placement tests. Finally, a 
large proportion of Asian and Latino students reported a language other than English as their 
primary language, this was also believed to have affected their performance on the English 



^^Skills Testing and Disproportionate Impact: An Analysis of the 

Reading and Writing Test Performance of Students in the San Diego Community 
College District. SDCCD Research and Planning, Spring, 1991 



plaQement tests. 

Table 11 below compares locally obtained average APS Reading and Writing 
placement test scores for groups by sex, ethnicity, and age. Predictor scores (Predictor = high 
school GPA plus recategorized APS Reading and Writing Placement) means are also 
compared. 



Table 11 

APS Reading and Writing and Predictor Means by Sex, Ethnic, and Age Groupings 



Group 


APS 

Reading 

(SD) 


APS 

Writing 

(SD) 


N 


Predictor 

(SD) 


Predictor 

Reading 

(SD) 


Predictor 

Writing 

(SD) 


Males 


18.5 


21.7 


2091 


13.3 


6.6 


6.7 




(7.6) 


(7.0) 




(2.8) 


(1.6) 


. (1.4) 


Females 


17.3 


22.3 


2188 


13.3 


6.4 


6.9 




(6.9) 


(7.0) 




(2.8) 


(1.6) 


(1.4) 


Whites 


21.6 


25.5 


1419 


14.5 


7.1 


7.3 




(6.9) 


(6.4) 




(2.7) 


(1.6) 


(1.4) 


Nonwhites 


16.6 


20.0 


2754 


12.6 


6.1 


6.5 




(6.7) 


(6.5) 




(2.6) 


(1.5) 


(1.3) 


Age 17 & 


20.5 


25.5 


57 


15.2 


7.4 


7.7 


Under 


(7.4) 


(6.5) 




(2.7) 


(1.6) 


(1.3) 


18-21 


17.6 


22.4 


2697 


13.5 


6.5 


6.9 




(6.9) 


(6.8) 




(2.8) 


(1.6) 


(1.3) 


22-25 


19.5 


22.3 


589 


13.3 


6.7 


6.6 




(7.3) 


(6.9) 




(2.8) 


(1.6) 


(1.4) 


26 & over 


17.4 


20.5 


979 


12.7 


6.3 


6.4 




(8.0) 


(7.6) 




(3.0) 


(1.7) 


(1.5) 


Total 


17.8 


22.0 


4322 


13.3 


6.5 


6.8 




(7.3) 


(7.0) 




(2.9) 


(1.6) 


(1.4) 



For the APS scores and Predictor scores, the ethnic, gender, and age differences are 
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statistically significant, although in the case of gender, the differences are the smallest. 

Whites score significantly higher than non- whites, and younger students tend to perform 
somewhat better on all measures than do older students. In the case of gender however, the 
proportion of variance accounted for by this variable is small. The eta-square statistic is .009, 
indicating that less than 1 % of the variance in the case of the Reading test is explained by 
gender. The proportion of variance explained by gender for the Writing placement test is also 
less than 1 %. Thus, although the differences are statistically significant, the proportion of 
variance accounted for by gender is minimal. Over 98% of the variance found between men 
and women is not explained by differences in gender. The ethnic differences (coded as 
white/nonwhite) are also statistically significant. As pointed out in the Disproportionate 
Impact study however, much of the observed differences between groups was attributable to 
differences in preparation ahd prior academic achievement in high school. This analysis is 
also complicated by the large number of non-native English speakers electing to take the 
English placement tests rather than the CELS A test for non-native speakers. Prior analysis has 
shown that non-native speakers attempt fewer questions and have more difficulty answering the 
self-assessment questions than native speakers. This ESL group is also included largely in the 
nonwhite category. The eta-squared statistic for the white and nonwhite comparison groups is 
.14, indicating that about 14% of the variance in Reading and the Writing score was explained 
by ethnic grouping. The proportion of variance explained by age grouping was 1% for both 
the Reading and Writing tests. 

When the new predictor is analyzed however, the proportion of variance explained by 
student grouping drops. In the case of gender, the eta-squared drops to non-significance 
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indicated by a statistic of less than .0001. Ethnic differences also diminish. Where the 
proportion of variance in test score explained by ethnic grouping was approximately 14%, the 
eta-square for the predictor drops to 10% of the variance. This suggests that the new predictor 
helps to mitigate potential disproportionate impact of the assessment process through the 
addition and weighting of high school GPA. The proportion of variance accounted for age 
grouping also drops using the new predictor. The eta-square statistic shows that less than 2% 
of the variance is explained by age category. Thus it appears that the new predictor does 
substantially mitigate potential disproportionate impact in the case of ethnic grouping, and 
reduces it to non-significance for sex and age groupings. 

Similar comparisons to monitor potential test bias or disproportionate impact were 
conducted using two predictors that are derived from the English 101 predictor. These scales, 
titled Predictor Reading and Predictor Writing in the preceding table, were developed to refer 
students scoring below the 101 level to one or both of the Reading and Writing curricular 
tracks in the English department. As described above, these scales are simply the respective 
placement test (e.g., Reading Test for reading track (English 55 and 56), Writing Test for 
writing track (English 50 and 51), and are used to refer students who score below the 
recommended cut off of 15 on the English 101 Predictor score. These new scales also 
mitigate potential disproportionate impact by reducing the eta-square statistic to less than 10% 
of the variance explained by ethnic grouping for Reading, and approximately 8% of the 
variance for Writing. At the lower levels, the new predictors also mitigate the potentially 
biasing effects of testing. 



O 
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Score Grade Correlations for APS Placement Tests and English Predictor by Ethnic 
Grouping. To further analyze the potential for mitigating disproportionate impact, separate 
correlational analyses were conducted for the white-non- white student groupings. This was 
done because the largest differences were observed for these two groups in terms of test and 
predictor scores. The following table summarizes these coefficients. The results suggest that 
when the new predictors are employed that include high school GPA, the strength of the 
correlation coefficient improves for both groupings. The correlations for the non- white 
students is on par with that of the white students, and in some cases is slightly higher. In all 
cases the correlations are statistically significant. 
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Table 12 

Score-Grade Correlations for APS Placement Tests by White-Non White Groupings 



Measure 


White 

GPA 


Nonwhite 

GPA 


APS 

Reading 

(N) 

Sig. 


.1745 

(850) 

P=.000 


. 1354 
( 1566) 
P= .000 


APS 

Writing 

(N) 

Sig. 


.2151 
(850) 
P=. 000 


. 1830 
( 1567) 
P= .000 


Reading 

Predictor 

(N) 

Sig. 


.2265 
(807) 
P=. 000 


. 2282 
( 1440) 
P= .000 


Writing 
Predictor 
. (N) 

Sig. 


.2477 
(807) 
P=. 000 


.2598 
( 1440) 
P= .000 


101 

Predictor 

(N) 

Sig. 


.2543 

(807) 

P=.000 


.2623 
( 1440) 
P= .000 



Nearly all correlations are comparable in value for the white/non white groupings. 

Thus both the APS and the three Predictors (English 101, Reading and Writing for below 101 
are significantly correlated with course performance for these two groups. Similar 
correlational analyses were conducted for gender and age groupings, and a similar pattern of 
correlations were found. The new Predictors for English 101, and below freshman 
composition improve the observed coefficients for both men and women, and for both younger 
and older students with little appreciable differences between these various groupings on the 
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dependent variable of final grade. Also, extensive analyses of potential disproportionate 
impact and test bias are contained in a technical appendix to this report and is bound 
separately. In this technical appendix data were analyzed using age, race, ethnic, and primary 
language of the fall, 1994 tested population. This report is included in the SDCCD Test 
Validation portfolio available for review in the Research and Planning office and at the 
colleges. 

Fairness in Placement: Alternative Evidence. As described in previous reports done by the 
Research office, student surveys analyzed by ethnic and gender grouping show few differences 
in terms of perceived fairness and equity of the assessment and placement process. The 
special Matriculation Surveys conducted for the campuses as well as districtwide suggest that 
students vary little in their satisfaction with regard to the assessment process by either ethnic 
or gender grouping. 

Cut and Placement Scores 
Local Studies 

The primary goal for establishing a preliminary cut score was to achieve the highest 
success rate while still allowing the greatest number of potentially eligible students to enroll in 
English 101. Additionally it was sought to maximize the proportion of correctly placed 
students compared to the baseline success rate while still maintaining access for the greatest 
number of students. Table 13 below shows various statistics for a range of plausible cut 

^^Student Equity and Satisfaction with Matriculation 
Services in the SDCCD, SDCCD Research and Planning, 1995. 
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scores for English 101. In this analyses, success is defined as receiving a grade of A, B, C, 
or Credit in the course. Non-success included all other transcript notations except drop. As 
the second and third columns show, cut scores in the range of 14 to 15 would result in 
approximately 40% of eligible students passing and from approximately 7% to 12% of 
ineligible students passing. These cutoffs would also place the most students accurately 
(approximately 58%). In consultation with the English department leadership, a score of 15 
on the freshman composition predictor was selected as the tentative skill level recommended 
for placement into English 101. Again, a caveat to this study is that these data are 
retrospective data. Students had been pre-sorted under a prior placement scheme, thus a large 
proportion of students may have been screened out due to ineligibility under the existing 
system. This pre-sorting has the effect of complicating interpretation of cut score tables 
because of the truncation of the range of abilities available for analysis. 



Table 13 

Cut Score Statistics for English 101 



Predictor 

Score 


% 

Ineligible 

Passing 


% 

Eligible 

Passing 


Correctly 

Placed 


Adds 

to 

Base 

line 


Number 

Eligible 


% 

Eligible 


11 


2.9 


47.3 


50.2 


0.0 


783 


94 . 1 


12 


3.3 


46.9 


51.5 


1.3 


721 


92.1 


13 


4.3 


45.8 


52.6 


2.5 


696 


88.9 


14 


6.9 


43.3 


55.4 


5.2 


634 


81.0 


15 


11.7 


38.4 


57.7 


7.6 


540 


69.0 


16 


19.3 


30.9 


58.5 


8.3 


416 


53.1 


17 


28.5 


21.7 


56.7 


6.5 


286 


36.5 
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The baseline rate of passing English 56 was approximately 52% (N=772) for this validation 
sample (i.e., without a cut score based on this predictor variable, 52% of these students passed 
English 56 with a grade of C or better. Cut scores of 15 and 16 raise the success rate by the 
largest margin. However a cut score of 16 dramatically reduces the proportion of students 
eligible for freshman composition. (Approximately 70% are eligible at a score of 15, while 
about 50% are eligible at a score of 16). A cut score of 15 places almost 60% of the sample 
correctly, with a corresponding increase of 8% to the baseline rate of success. The trade-off 
between proportion eligible and baseline improvements to the success rate begin to diminish at 
the upper levels as illustrated in the table when the cut score increases. 

Thus recommendations were made to the English department chairs to test a score of 
15 for referral to freshman composition. This recommendation was implemented and is 
currently being tested with new students. Future reports will continue to monitor this 
placement recommendation. 

Cut Scores for Below Freshman Composition 

As described earlier in the development of the English 101 predictor, separate scales 
were derived from the English 101 predictor to refer students to needed foundational courses 
in either Reading, Writing, or both to prepare for freshman composition. Cut score statistics 
were also produced for English 56 and English 51 to determine cut scores and baseline 
improvements for success rates similar to those produced for English 101 in the above table. 
English 50 and 55 are basic skills courses for which there is not a prerequisite for entry. The 
caveats that applied to the interpretation of the cut score tables earlier also apply to the 
analysis that follows. Students in the validation sample had been pre-sorted using an existing 
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placement scheme. These retrospective data then reflect the limitations of using these statistics 
when an existing assessment system is used to place students. These tables must be interpreted 
with care. The cut score tables for English 51 (Writing Course) and English 56 (Reading 
Course) are reproduced below. 



Table 14 

Cut Score Statistics for English 56 Using Reading Predictor 



Predictor 

Score 


% 

Ineligible 

Passing 


% 

Eligible 

Passing 


Correctly 

Placed 


Adds 

to 

Base 

line 


Number 

Eligible 


% 

Eligible 


4.5 


22.0 


53 . 8 


54.9 


2.6 


736 


93.3 


5.0 


29.0 


54.6 


56.0 


3.7 


703 


91.1 


5.5 


41.2 


55.6 


56.5 


4.2 


595 


77.1 


6.0 


44.9 


57.2 


56.5 


4.2 


467 


60.5 


6.5 


50.1 


56.4 


52.1 


-..02 


273 


35.4 


7.0 


51.7 


56.6 


49.4 


-2.9 


99 


12.8 



This validation sample for English 56, as with the other data in this report use 
retrospective data obtained from the fall, 1994 tested population in the SDCCD. The number 
of cases available for analysis in the English 56 cut score validation study is 772. For the 10.5 
Reading Predictor scale cut score statistics are produced above. A range of plausible cut scores 
is shown in the table above for English 56. A cut score of from 5.5 to 6.0 resulted in the 
largest additions to the baseline while still allowing for access. In consultation with the 
English department and student services leadership a tentative cut score of 6.0 was 
recommended for referral to English 56. This was selected because it added to the, baseline 



rate, but also out of concerns that students indicating the highest high school GPA (i.e., a 
score of 3.5) would be eligible for English 56 even with very low test scores. It was the 
judgement of the teaching faculty that the distributions shown here and based on simulations 
using the new model would be the most prudent. Students scoring below the freshman 
composition level would be directed into English 56 with a minimum score of 6.0 on the 
Reading Predictor scale. The system is currently in place and will be monitored for its impact 
on student eligibility and success rates. 

Table 15 below contains cut score statistics for the Writing Predictor scale for students 
falling below the freshman composition level on the Predictor who should enroll to improve 
writing skills in preparation for college level English. A range of plausible cut scores based 
on the 10.5 Writing Predictor scale are shown below. A score of 6.5 results in the highest 
addition (approximately 8%) to the baserate of success for English 51 for this validation 
sample (N=523). Also the English department leadership felt as they did with the referrals to 
English 56, that students who indicated the highest GPA possible yet still scored very low on 
the Reading Placement test scale should be placed in the basic skills writing course, English 
50. Using their professional judgement, discipline faculty in English selected 6.5 as the 
tentative cut off for eligibility for English 51. 
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Table 15 

Cut Score Statistics for English 51 Using Writing Predictor 



Predictor 

Score 


% 

Ineligible 

Passing 


% 

Eligible 

Passing 


Correctly 

Placed 


Adds 

to 

Base 
. line 


Niixnber 

Eligible 


% 

Eligible 


5.0 


37.1 


50.8 


51.6 


1.6 


488 


93.3 


5.5 


36.1 


53.1 


55.0 


5.0 


426 


81.5 


6.0 


37.2 


55.7 


57.9 


7.9 


359 


68.6 


6.5 


42.8 


59.3 


58.1 


8.1 


226 


43.2 


7.0 


46.8 


60.9 


54.9 


4.9 


115 


22 . 0 


7.5 


47.6 


73.9 


54.3 


4.3 


46 


8.8 



This cut score for the Writing Predictor will be also be monitored for impact on the eligibility 
and success rates of students. 

Summary 

Using a variety of evidence from a diversity of sources including statistical analyses, 
student survey data, and faculty surveys, this study suggests that the APS Placement Tests 
have acceptable criterion-related validity for placing students in the English curriculum. 
Although it must be noted that this study did rely on retrospective data that used students in a 
sample that had been pre-sorted using a pre-existing placement system. Thus the results must 
be interpreted with caution. This retrospective study identified several variables that 
correlated significantly with success in English courses. Foremost among these were high 
school GPA, years of English completed, grade in last English class, and number of years out 
of school. Stepwise multiple regression procedures identified high school GPA as the 
strongest predictor along with the combined APS Reading and English Placement Test scores. 



To obtain the simplest and most understandable measure, and one that can be practically 
collected, it was decided to include high school GPA as part of a new predictor scale. To 
accomplish this, the Reading and Writing Placement tests were each recategorized into a 
seven point scale and added to the existing seven point scale for high school GPA. This new 
scale ranged from 3 to 21 with high school GPA counting for one-third of the total. The new 
distribution appeared approximately normal and the measures of central tendency tended to 
confirm the normality of the distribution. This English 101 Predictor scale was found to have 
higher correlations with the dependent variable of final grade in English than the APS tests did 
alone. Further it was found that high school GPA was not highly correlated with the 
placement tests scores. For placement below the freshman composition level, two scales 
derived from the English 101 predictor scale were tested. Both showed statistically significant 
correlations with the criterion variable of final grade. In the Reading and Writing Predictor 
scales, as with the Freshman Composition Predictor, high school GPA accounts for one-third 
of the total scale for a range of 10.5 points (Reading or Writing =1 + high school GPA 



=3.5). 



When monitored for disproportionate impact, it was found that the new measure 
mitigated the effects of student ethnicity, gender, and age in explaining variance in final grade. 
Of the three student groupings analyzed, ethnicity seemed to have the largest proportion of 
variance of final grade. Separate correlations were run for whites and non- whites and 
suggested that the correlations for non-white ethnic groupings were almost equivalent to those 
found for white students. Extensive analyses of potential disproportionate impact were also 
conducted and are included in a technical appendix bound separately from this report. 
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Cut score statistics and validation tables showed the proportion of eligible and 
ineligible students passing and the amount of improvement in the baseline success rates at 
several plausible cut scores. Discipline faculty selected a score of 15 on the English 101 
Predictor. For English 56 a score of 6.0 for eligibility was selected. On the Writing 
Predictor scale, a score of 6.5 was selected by the faculty as the new tentative cut scores. The 
Research office will continue to report and monitor the effects and impacts of these scales and 
cut scores on the eligibility and success rates of students. 
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Appendix A 



Descriptive Statistics of Sample and Score Distributions of APS and English 
Predictors based on Test Scores and High School GPA 



Distributions of Placements Using Modified Reading and Writing Scale with 
High School GPA Reduced to One Third of Respective Scales 



CO 




io 




Note: Under new multiple measures scheme, there are no ESL referrals based on APS test scores, high school GPA, and student survey responses. Previous 
system did include ESL referrals. The data for the APS (single measure scheme) does not include those eligible for ESL only. For the multiple measure scale 
data, all students are considered eligible for the two lowest level English courses. This may have the effect of increasing those eligible for English 50 and 55. 
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Frequency 




Std. Dev = .59 
Mean = 2.09 
N = 2901.00 



NEWHSGPA 





NEWHSGPA 



Valid 

Value Label Value Frequency Percent Percent 



.50 


41 


1.4 


1.4 


1.00 


148 


5.1 


5-.1 


1.50 


587 


20.2 


20.2 


2.00 


1038 


35.8 


35.8 


2.50 


682 


23.5 


23.5 


3.00 


348 


12.0 


12.0 


3.50 


57 


2.0 


2.0 


Total 


2901 


100.0 


100.0 



Hi-Res Chart # 4:Histogram of newhsgpa 



Mean 


2.094 


Std err 


.011 


Medicin 


Mode 


2.000 


Std dev 


.590 


Variance 


Maximum 


3.500 








Valid cases 


2901 


Missing cases 


0 






GO 



Cum 

Percent 

1.4 

6.5 
26.7 
62.5 
86.0 
98.0 

100.0 



2.000 

.348 



NEWENGL 











Valid 


Cum 


Value Label 


Value Frequency 


Percent 


Percent 


Percent 




2.00 


1 


.0 


.0 


.0 




2.50 


5 


.2 


.2 


.2 




3.00 


14 


.5 


.5 


.7 




3.50 


35 


1.2 


1.2 


1.9 




4.00 


82 


2.8 


2.8 


4.7 




4.50 


167 


5.8 


5.8 


10.5 




5.00 


309 


10.7 


10.7 


21.1 




5.50 


388 


13.4 


13.4 


34.5 




6.00 


577 


19.9 


19.9 


54.4 




6.50 


508 


17.5 


17.5 


71.9 




7.00 


499 


17.2 


17.2 


89.1 




7.50 


255 


8.8 


8.8 


97.9 




8.00 


51 


1.8 


1.8 


99.7 




8.50 


9 


.3 


.3 


100.0 




9.00 


1 


.0 


.0 


100.0 




Total 


2901 


100.0 


100.0 




Hi-Res Chart 


# 6: Histogram of newengl 










Mean 


6.067 Std err 


.019 


Median 


6.000 


Mode 


6 . 000 Std dev 


1.015 


Variance 


1.030 


Maximum 


9.000 










Valid cases 


2901 Missing cases. 


0 










Histogram 
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Histogram 
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NEWREAD 



Value Label 



Hi -Res Chart 

Mean 

Mode 

Maximum 



Valid cases 



O 




Valid Cum 



Value 


Frequency 


Percent 


Percent 


Percent 


1.50 


2 


.1 


.1 


. 1 


2.00 


2 


.1 


.1 


.1 


2.50 


17 


.6 


.6 


.7 


3.00 


47 


1.6 


1.6 


2.3 


3.50 


89 


3.1 


3.1 


5.4 


4.00 


195 


6.7 


6.7 


12.1 


4.50 


268 


9.2 


9.2 


21.4 


5.00 


383 


13.2 


13.2 


34.6 


5.50 


386 


13.3 


13.3 


.47.9 


6.00 


489 


16.9 


16.9 


64.7 


6.50 


456 


15.7 


15.7 


80 . 5 


7.00 


356 


12.3 


12.3 


92.7 


7.50 


140 


4.8 


4.8 


97.6 


8.00 


60 


2.1 


2.1 


99.6 


8.50 


9 


.3 


.3 


99.9 


9.00 


2 


.1 


.1 


100.0 


Total 


2901 


100.0 


100.0 





# 5: Histogram of newread 

5.702 Std err .022 Median 6.000 

6.000 Std dev 1.169 Variance 1.367 

9.000' 

2901 Missing cases 0 
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-> LIST 

-> VARIABLES=reading recredl writing recengl hsgpa newhsgpa newread newengl 
-> /CASES= FROM 1 TO 250 BY 1 
-> /FORMAT* WRAP UNNUMBERED . 




G5 



READING 


RECREDl 


WRITING 


RECENGl 


HSGPA 


NEWHSGPA 


NEWREAD 


NEWENGIi 


24.00 


5.00 


21.00 


4.00 


5.00 


2.50 


7.50 


6.50 


10.00 


2.00 


21.00 


4.00 


5.00 


2.50 


4.50 


6.50 


10.00 


2.00 


20.00 


4.00 


6.00 


3.00 


5.00 


7.00 


18.00 


4.00 


25.00 


5.00 


3.00 


1.50 


5.50 


6.50 


7.00 


2.00 


15.00 


3.00 


3.00 


1.50 


3.50 


4 . 50 


21.00 


5.00 


15.00 


3.00 


3.00 


1.50 


6.50 


4.50 


8 . 00 


2.00 


16.00 


3.00 


3.00 


1.50 


3.50 


4.50 


19.00 


4.00 


16.00 


3.00 


3.00 


1.50 


5.50 


4.50 


7.00 


2.00 


11.00 


2.00 


4.00 


2.00 


4.00 


4.00 


12.00 


3.00 


9.00 


2.00 


5.00 


2.50 


5.50 


4 . 50 


12.00 


3.00 


21.00 


4.00 


• 6.00 


3.00 


6.00 


7.00 


18.00 


4.00 


23.00 


4.00 


5.00 


2.50 


6.50 


6.50 


18.00 


4.00 


15.00 


3.00 


3.00 


1.50 


5.50 


4 . 50 


18.00 


4.00 


24.00 


5 . 00 


2.00 


1.00 


5.00 


6.00 


5.00 


1.00 


9.00 


2.00 


6.00 


3.00 . 


4.00 


5.00 


14.00 


3 . 00 


17.00 


3.00 


6.00 


3.00 


6.00 


6.00 


22.00 


5.00 


23.00 


4.00 


4.00 


2.00 


7.00 


6.00 


18.00 


4.00 


21.00 


4.00 


5.00 


2.50 


6.50 


6.50 


16.00 


4.00 


20.00 


4.00 


6.00 


3.00 


7.00 


7.00 


15.00 


3.00 


22.00 


4.00 


5.00 


2.50 


5.50 


6.50 


14.00 


3.00 


27.00 


5.00 


5.00 


2.50 


5.50 


7.50 


17.00 


4.00 


26.00 


5.00 


5.00 


2.50 


6.50 


7.50 


11.00 


3.00 


21.00 


4.00 


4.00 


2.00 


5.00 


6.00 


20.00 


4.00 


24.00 


5.00 


4.00 


2.00 


6.00 


7.00 


5.00 


1.00 


13.00 


3.00 


6.00 


3.00 


4.00 


6.00 


14.00 


3.00 


27.00 


5.00 


6.00 


3.00 


6.00 


8.00 


14.00 


3.00 


8.00 


2.00 


4 . 00 


2.00 


^ 5,00 


4 . 00 


10.00 


2.00 


21.00 


4.00 


5.00 


2.50 


4.50 


6.50 


12.00 


3.00 


14.00 


3.00 


4.00 


2.00 


5.00 


5.00 


21.00 


5.00 


16.00 


3.00 


2.00 


1.00 


6.00 


4.00 


15.00 


3.00 


18.00 


4.00 


4 . 00 


2.00 
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